{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Outline:  \n",
    "In this tutorial, we are going to discuss about implementation of BERT model purposed Nov. 2018.  \n",
    "We will mainly focus on 4 aspects:  \n",
    "\n",
    "1. What is BERT and word embedding?  \n",
    "2. What is a tokenizer and contextual level embedding?  \n",
    "3. How to get word, sentence, and paragraph embeddings from token level embeddings?  \n",
    "4. How to fine tune your own BERT model?    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. What is BERT and word embedding?\n",
    "\"This week, we **open sourced** a new technique for NLP **pre-training** called Bidirectional Encoder Representations from Transformers, or BERT. With this release, anyone in the world can train their own **state-of-the-art** question answering system (or a variety of other models) in about 30 minutes on a single Cloud TPU, or in a few hours using a single GPU.\" - [Google AI Blog, Nov.2.2018](https://ai.googleblog.com/2018/11/open-sourcing-bert-state-of-art-pre.html)  \n",
    "\n",
    "\n",
    "<img src=https://4.bp.blogspot.com/-iQZIsE3lbVY/W9i8Tc-F7RI/AAAAAAAADfU/DrxjBoDfqrwe6GJUxENqWuzQ0IPlgT3TgCLcBGAs/s1600/image3.png width=\"600\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Motivitions:\n",
    " - The first deeply bidirectional contextual embedding. (vs Glove)\n",
    " - Data gap for Natural Language Processing tasks.  \n",
    " - Easy to use and fine tune for you own project to achieve state of the art."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. What is a tokenizer and what is contextual embedding?\n",
    "From word to embedding vector:  \n",
    "word -> tokens -> ids -> tensors -> hidden state vectors -> embeddings  \n",
    "\n",
    "Contextua = 1 to many  \n",
    "Non-contextual = 1 to 1\n",
    "  \n",
    "   \n",
    "link pip install torch: https://pytorch.org/   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ! pip install pytorch_pretrained_bert"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True\n",
      "GeForce RTX 2080 Ti\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "print(torch.cuda.is_available())\n",
    "print(torch.cuda.get_device_name(0))\n",
    "import torch\n",
    "from pytorch_pretrained_bert import BertTokenizer, BertModel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "RETRAINED_POSITIONAL_EMBEDDINGS_SIZES = {  \n",
    "    'bert-base-uncased': 512,  \n",
    "    'bert-large-uncased': 512,  \n",
    "    'bert-base-cased': 512,  \n",
    "    'bert-large-cased': 512,  \n",
    "    'bert-base-multilingual-uncased': 512,  \n",
    "    'bert-base-multilingual-cased': 512,  \n",
    "    'bert-base-chinese': 512,  \n",
    "    'bert-base-german-cased': 512,  \n",
    "    'bert-large-uncased-whole-word-masking': 512,  \n",
    "    'bert-large-cased-whole-word-masking': 512,  \n",
    "    'bert-large-uncased-whole-word-masking-finetuned-squad': 512,  \n",
    "    'bert-large-cased-whole-word-masking-finetuned-squad': 512,  \n",
    "    'bert-base-cased-finetuned-mrpc': 512,  \n",
    "}  \n",
    "\n",
    "from https://github.com/huggingface/pytorch-transformers/blob/df9d6effae43e92761eb92540bc45fac846789ee/pytorch_transformers/tokenization_bert.py#L86"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wrong sentence:\n",
      " [CLS]Seahorses dreamed about word embeddings and artificial intelligence[SEP]\n",
      "['[', 'cl', '##s', ']', 'sea', '##horse', '##s', 'dreamed', 'about', 'word', 'em', '##bed', '##ding', '##s', 'and', 'artificial', 'intelligence', '[', 'sep', ']'] \n",
      "\n",
      "Right sentence:\n",
      " [CLS] Seahorses dreamed about word embeddings and artificial intelligence [SEP]\n",
      "['[CLS]', 'sea', '##horse', '##s', 'dreamed', 'about', 'word', 'em', '##bed', '##ding', '##s', 'and', 'artificial', 'intelligence', '[SEP]']\n"
     ]
    }
   ],
   "source": [
    "# word -> tokens\n",
    "raw_text_a = \"Seahorses dreamed about word embeddings and artificial intelligence\"\n",
    "raw_text_b = \"Seahorse is a smart animal\"\n",
    "text_a = \"[CLS] \" + raw_text_a + \" [SEP]\" # notice the space\n",
    "text_b = raw_text_b + \" [SEP]\" # don't need [CLS] for the second sentence\n",
    "tokens_a = tokenizer.tokenize(text_a)\n",
    "tokens_b = tokenizer.tokenize(text_b)\n",
    "\n",
    "wrong_tokens = tokenizer.tokenize(\"[CLS]\" + raw_text_a + \"[SEP]\")\n",
    "print(\"Wrong sentence:\\n\", \"[CLS]\" + raw_text_a + \"[SEP]\")\n",
    "print(wrong_tokens, \"\\n\")\n",
    "print(\"Right sentence:\\n\", text_a)\n",
    "print(tokens_a) # notice the word seahorses and embedding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['[PAD]', '[unused0]', '[unused1]', '[unused2]', '[unused3]']\n",
      "['journey', 'converted', 'provincial', 'painted', 'hearing', 'aren', 'bands', 'negative', 'aside', 'wondered', 'knight', 'lap', 'survey', 'ma', '##ow']\n"
     ]
    }
   ],
   "source": [
    "# curious about tokenizer \n",
    "print(list(tokenizer.vocab.keys())[:5])\n",
    "print(list(tokenizer.vocab.keys())[4990:5005])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['[CLS]', 'sea', '##horse', '##s', 'dreamed', 'about', 'word', 'em', '##bed', '##ding', '##s', 'and', 'artificial', 'intelligence', '[SEP]']\n",
      "['sea', '##horse', 'is', 'a', 'smart', 'animal', '[SEP]']\n"
     ]
    }
   ],
   "source": [
    "print(tokens_a)\n",
    "print(tokens_b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tokens: ['[CLS]', 'sea', '##horse', '##s', 'dreamed', 'about', 'word', 'em', '##bed', '##ding', '##s', 'and', 'artificial', 'intelligence', '[SEP]', 'sea', '##horse', 'is', 'a', 'smart', 'animal', '[SEP]']\n",
      "type_ids: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1]\n"
     ]
    }
   ],
   "source": [
    "# word -> tokens -> ids -> hidden states -> embeddings\n",
    "\n",
    "tokens = []\n",
    "input_type_ids = []\n",
    "# masks for segment, 0 for the first sentence, 1 for the second sentence.\n",
    "# use 1 if there's only one sentence.\n",
    "\n",
    "for token in tokens_a:\n",
    "    tokens.append(token)\n",
    "    input_type_ids.append(0)\n",
    "\n",
    "if tokens_b:\n",
    "    for token in tokens_b:\n",
    "        tokens.append(token)\n",
    "        input_type_ids.append(1)\n",
    "        \n",
    "print(\"tokens:\", tokens)   \n",
    "print(\"type_ids:\", input_type_ids)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('[CLS]', 101)\n",
      "('sea', 2712)\n",
      "('##horse', 23024)\n",
      "('##s', 2015)\n",
      "('dreamed', 13830)\n",
      "('about', 2055)\n",
      "('word', 2773)\n",
      "('em', 7861)\n",
      "('##bed', 8270)\n",
      "('##ding', 4667)\n",
      "('##s', 2015)\n",
      "('and', 1998)\n",
      "('artificial', 7976)\n",
      "('intelligence', 4454)\n",
      "('[SEP]', 102)\n",
      "('sea', 2712)\n",
      "('##horse', 23024)\n",
      "('is', 2003)\n",
      "('a', 1037)\n",
      "('smart', 6047)\n",
      "('animal', 4111)\n",
      "('[SEP]', 102)\n"
     ]
    }
   ],
   "source": [
    "# tokens -> ids\n",
    "input_ids = tokenizer.convert_tokens_to_ids(tokens)\n",
    "for pair in zip(tokens, input_ids):\n",
    "    print(pair)\n",
    "# notice the case ---> uncased"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[101, 2712, 23024, 2015, 13830, 2055, 2773, 7861, 8270, 4667, 2015, 1998, 7976, 4454, 102, 2712, 23024, 2003, 1037, 6047, 4111, 102]\n",
      "[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]\n",
      "[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1]\n",
      "\n",
      "[101, 2712, 23024, 2015, 13830, 2055, 2773, 7861, 8270, 4667, 2015, 1998, 7976, 4454, 102, 2712, 23024, 2003, 1037, 6047, 4111, 102, 0, 0, 0, 0, 0, 0, 0, 0]\n",
      "[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0]\n",
      "[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0]\n"
     ]
    }
   ],
   "source": [
    "# padding\n",
    "seq_length = 30 # max allowed length & padding length for each pair of sentences. 512\n",
    "input_mask = [1] * len(input_ids)\n",
    "print(input_ids)\n",
    "print(input_mask)\n",
    "print(input_type_ids)\n",
    "while len(input_ids) < seq_length:\n",
    "    input_ids.append(0)\n",
    "    input_mask.append(0)\n",
    "    input_type_ids.append(0)\n",
    "    \n",
    "print()\n",
    "print(input_ids)\n",
    "print(input_mask)\n",
    "print(input_type_ids)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "BertModel(\n",
       "  (embeddings): BertEmbeddings(\n",
       "    (word_embeddings): Embedding(30522, 768, padding_idx=0)\n",
       "    (position_embeddings): Embedding(512, 768)\n",
       "    (token_type_embeddings): Embedding(2, 768)\n",
       "    (LayerNorm): BertLayerNorm()\n",
       "    (dropout): Dropout(p=0.1)\n",
       "  )\n",
       "  (encoder): BertEncoder(\n",
       "    (layer): ModuleList(\n",
       "      (0): BertLayer(\n",
       "        (attention): BertAttention(\n",
       "          (self): BertSelfAttention(\n",
       "            (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (dropout): Dropout(p=0.1)\n",
       "          )\n",
       "          (output): BertSelfOutput(\n",
       "            (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (LayerNorm): BertLayerNorm()\n",
       "            (dropout): Dropout(p=0.1)\n",
       "          )\n",
       "        )\n",
       "        (intermediate): BertIntermediate(\n",
       "          (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "        )\n",
       "        (output): BertOutput(\n",
       "          (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "          (LayerNorm): BertLayerNorm()\n",
       "          (dropout): Dropout(p=0.1)\n",
       "        )\n",
       "      )\n",
       "      (1): BertLayer(\n",
       "        (attention): BertAttention(\n",
       "          (self): BertSelfAttention(\n",
       "            (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (dropout): Dropout(p=0.1)\n",
       "          )\n",
       "          (output): BertSelfOutput(\n",
       "            (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (LayerNorm): BertLayerNorm()\n",
       "            (dropout): Dropout(p=0.1)\n",
       "          )\n",
       "        )\n",
       "        (intermediate): BertIntermediate(\n",
       "          (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "        )\n",
       "        (output): BertOutput(\n",
       "          (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "          (LayerNorm): BertLayerNorm()\n",
       "          (dropout): Dropout(p=0.1)\n",
       "        )\n",
       "      )\n",
       "      (2): BertLayer(\n",
       "        (attention): BertAttention(\n",
       "          (self): BertSelfAttention(\n",
       "            (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (dropout): Dropout(p=0.1)\n",
       "          )\n",
       "          (output): BertSelfOutput(\n",
       "            (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (LayerNorm): BertLayerNorm()\n",
       "            (dropout): Dropout(p=0.1)\n",
       "          )\n",
       "        )\n",
       "        (intermediate): BertIntermediate(\n",
       "          (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "        )\n",
       "        (output): BertOutput(\n",
       "          (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "          (LayerNorm): BertLayerNorm()\n",
       "          (dropout): Dropout(p=0.1)\n",
       "        )\n",
       "      )\n",
       "      (3): BertLayer(\n",
       "        (attention): BertAttention(\n",
       "          (self): BertSelfAttention(\n",
       "            (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (dropout): Dropout(p=0.1)\n",
       "          )\n",
       "          (output): BertSelfOutput(\n",
       "            (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (LayerNorm): BertLayerNorm()\n",
       "            (dropout): Dropout(p=0.1)\n",
       "          )\n",
       "        )\n",
       "        (intermediate): BertIntermediate(\n",
       "          (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "        )\n",
       "        (output): BertOutput(\n",
       "          (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "          (LayerNorm): BertLayerNorm()\n",
       "          (dropout): Dropout(p=0.1)\n",
       "        )\n",
       "      )\n",
       "      (4): BertLayer(\n",
       "        (attention): BertAttention(\n",
       "          (self): BertSelfAttention(\n",
       "            (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (dropout): Dropout(p=0.1)\n",
       "          )\n",
       "          (output): BertSelfOutput(\n",
       "            (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (LayerNorm): BertLayerNorm()\n",
       "            (dropout): Dropout(p=0.1)\n",
       "          )\n",
       "        )\n",
       "        (intermediate): BertIntermediate(\n",
       "          (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "        )\n",
       "        (output): BertOutput(\n",
       "          (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "          (LayerNorm): BertLayerNorm()\n",
       "          (dropout): Dropout(p=0.1)\n",
       "        )\n",
       "      )\n",
       "      (5): BertLayer(\n",
       "        (attention): BertAttention(\n",
       "          (self): BertSelfAttention(\n",
       "            (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (dropout): Dropout(p=0.1)\n",
       "          )\n",
       "          (output): BertSelfOutput(\n",
       "            (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (LayerNorm): BertLayerNorm()\n",
       "            (dropout): Dropout(p=0.1)\n",
       "          )\n",
       "        )\n",
       "        (intermediate): BertIntermediate(\n",
       "          (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "        )\n",
       "        (output): BertOutput(\n",
       "          (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "          (LayerNorm): BertLayerNorm()\n",
       "          (dropout): Dropout(p=0.1)\n",
       "        )\n",
       "      )\n",
       "      (6): BertLayer(\n",
       "        (attention): BertAttention(\n",
       "          (self): BertSelfAttention(\n",
       "            (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (dropout): Dropout(p=0.1)\n",
       "          )\n",
       "          (output): BertSelfOutput(\n",
       "            (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (LayerNorm): BertLayerNorm()\n",
       "            (dropout): Dropout(p=0.1)\n",
       "          )\n",
       "        )\n",
       "        (intermediate): BertIntermediate(\n",
       "          (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "        )\n",
       "        (output): BertOutput(\n",
       "          (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "          (LayerNorm): BertLayerNorm()\n",
       "          (dropout): Dropout(p=0.1)\n",
       "        )\n",
       "      )\n",
       "      (7): BertLayer(\n",
       "        (attention): BertAttention(\n",
       "          (self): BertSelfAttention(\n",
       "            (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (dropout): Dropout(p=0.1)\n",
       "          )\n",
       "          (output): BertSelfOutput(\n",
       "            (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (LayerNorm): BertLayerNorm()\n",
       "            (dropout): Dropout(p=0.1)\n",
       "          )\n",
       "        )\n",
       "        (intermediate): BertIntermediate(\n",
       "          (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "        )\n",
       "        (output): BertOutput(\n",
       "          (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "          (LayerNorm): BertLayerNorm()\n",
       "          (dropout): Dropout(p=0.1)\n",
       "        )\n",
       "      )\n",
       "      (8): BertLayer(\n",
       "        (attention): BertAttention(\n",
       "          (self): BertSelfAttention(\n",
       "            (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (dropout): Dropout(p=0.1)\n",
       "          )\n",
       "          (output): BertSelfOutput(\n",
       "            (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (LayerNorm): BertLayerNorm()\n",
       "            (dropout): Dropout(p=0.1)\n",
       "          )\n",
       "        )\n",
       "        (intermediate): BertIntermediate(\n",
       "          (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "        )\n",
       "        (output): BertOutput(\n",
       "          (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "          (LayerNorm): BertLayerNorm()\n",
       "          (dropout): Dropout(p=0.1)\n",
       "        )\n",
       "      )\n",
       "      (9): BertLayer(\n",
       "        (attention): BertAttention(\n",
       "          (self): BertSelfAttention(\n",
       "            (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (dropout): Dropout(p=0.1)\n",
       "          )\n",
       "          (output): BertSelfOutput(\n",
       "            (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (LayerNorm): BertLayerNorm()\n",
       "            (dropout): Dropout(p=0.1)\n",
       "          )\n",
       "        )\n",
       "        (intermediate): BertIntermediate(\n",
       "          (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "        )\n",
       "        (output): BertOutput(\n",
       "          (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "          (LayerNorm): BertLayerNorm()\n",
       "          (dropout): Dropout(p=0.1)\n",
       "        )\n",
       "      )\n",
       "      (10): BertLayer(\n",
       "        (attention): BertAttention(\n",
       "          (self): BertSelfAttention(\n",
       "            (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (dropout): Dropout(p=0.1)\n",
       "          )\n",
       "          (output): BertSelfOutput(\n",
       "            (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (LayerNorm): BertLayerNorm()\n",
       "            (dropout): Dropout(p=0.1)\n",
       "          )\n",
       "        )\n",
       "        (intermediate): BertIntermediate(\n",
       "          (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "        )\n",
       "        (output): BertOutput(\n",
       "          (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "          (LayerNorm): BertLayerNorm()\n",
       "          (dropout): Dropout(p=0.1)\n",
       "        )\n",
       "      )\n",
       "      (11): BertLayer(\n",
       "        (attention): BertAttention(\n",
       "          (self): BertSelfAttention(\n",
       "            (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (dropout): Dropout(p=0.1)\n",
       "          )\n",
       "          (output): BertSelfOutput(\n",
       "            (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (LayerNorm): BertLayerNorm()\n",
       "            (dropout): Dropout(p=0.1)\n",
       "          )\n",
       "        )\n",
       "        (intermediate): BertIntermediate(\n",
       "          (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "        )\n",
       "        (output): BertOutput(\n",
       "          (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "          (LayerNorm): BertLayerNorm()\n",
       "          (dropout): Dropout(p=0.1)\n",
       "        )\n",
       "      )\n",
       "    )\n",
       "  )\n",
       "  (pooler): BertPooler(\n",
       "    (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "    (activation): Tanh()\n",
       "  )\n",
       ")"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "# Load pre-trained model (weights)\n",
    "model = BertModel.from_pretrained('bert-base-uncased')\n",
    "model = model.cuda()\n",
    "# Put the model in \"evaluation\" mode, meaning feed-forward operation.\n",
    "model.eval()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([30, 1])\n",
      "torch.Size([30, 1])\n",
      "torch.Size([30, 1])\n"
     ]
    }
   ],
   "source": [
    "# Predict hidden states features for each layer\n",
    "with torch.no_grad():\n",
    "    # ids -> hidden state vectors\n",
    "    input_tensor = torch.LongTensor(input_ids).cuda().view(-1,1)\n",
    "    input_mask = torch.LongTensor(input_mask).cuda().view(-1,1)\n",
    "    input_type_ids = torch.LongTensor(input_type_ids).cuda().view(-1,1)\n",
    "    \n",
    "    print(input_tensor.shape)\n",
    "    print(input_mask.shape)  \n",
    "    print(input_type_ids.shape)\n",
    "    encoded_layers, _ = model(input_tensor, token_type_ids=input_type_ids, attention_mask=input_mask)\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'list'> 12\n",
      "<class 'torch.Tensor'> torch.Size([30, 1, 768])\n",
      "torch.Size([30, 1])\n"
     ]
    }
   ],
   "source": [
    "print(type(encoded_layers), len(encoded_layers))\n",
    "print(type(encoded_layers[0]), encoded_layers[0].shape)\n",
    "print(input_tensor.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0th layer with shape = torch.Size([30, 1, 768])\n",
      "1th layer with shape = torch.Size([30, 1, 768])\n",
      "2th layer with shape = torch.Size([30, 1, 768])\n",
      "3th layer with shape = torch.Size([30, 1, 768])\n",
      "4th layer with shape = torch.Size([30, 1, 768])\n",
      "5th layer with shape = torch.Size([30, 1, 768])\n",
      "6th layer with shape = torch.Size([30, 1, 768])\n",
      "7th layer with shape = torch.Size([30, 1, 768])\n",
      "8th layer with shape = torch.Size([30, 1, 768])\n",
      "9th layer with shape = torch.Size([30, 1, 768])\n",
      "10th layer with shape = torch.Size([30, 1, 768])\n",
      "11th layer with shape = torch.Size([30, 1, 768])\n"
     ]
    }
   ],
   "source": [
    "for i,layer in enumerate(encoded_layers):\n",
    "    print(f\"{i}th layer with shape = {layer.shape}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Now, what do we do with these hidden states?\n",
    "\n",
    "(Image from [Jay Allamar](http://jalammar.github.io/illustrated-bert/)'s blog)\n",
    "\n",
    "\n",
    "![alt text](http://jalammar.github.io/images/bert-feature-extraction-contextualized-embeddings.png)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CLS] Seahorses dreamed about word embeddings and artificial intelligence [SEP] Seahorse is a smart animal [SEP]\n",
      "torch.Size([30, 1, 768])\n"
     ]
    }
   ],
   "source": [
    "# to get the token embedding vector, we can sum the last four\n",
    "print(text_a, text_b)\n",
    "sum_last_four = torch.sum(torch.stack(encoded_layers[-4:]), dim=0)\n",
    "print(sum_last_four.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([120, 1, 768])\n"
     ]
    }
   ],
   "source": [
    "print(torch.cat(encoded_layers[-4:]).shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. From tokens to everything.\n",
    "Now we have embedding vectors for each token, how do we use them to get embedding vector for word, sentence, and paragraph?   \n",
    " - sum\n",
    " - average\n",
    "   \n",
    "Getting sentence embedding is easy. Getting word embedding can be tricky."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape of tokens embedding: torch.Size([30, 1, 768])\n",
      "Shape of sentence embedding after averaging:  torch.Size([30, 768])\n"
     ]
    }
   ],
   "source": [
    "# token -> sentence \n",
    "print(\"Shape of tokens embedding:\", sum_last_four.shape)\n",
    "print(\"Shape of sentence embedding after averaging: \",torch.mean(sum_last_four, 1).shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Seahorses', 'dreamed', 'about', 'word', 'embeddings', 'and', 'artificial', 'intelligence', 'Seahorse', 'is', 'a', 'smart', 'animal']\n",
      "['[CLS]', 'sea', '##horse', '##s', 'dreamed', 'about', 'word', 'em', '##bed', '##ding', '##s', 'and', 'artificial', 'intelligence', '[SEP]', 'sea', '##horse', 'is', 'a', 'smart', 'animal', '[SEP]']\n"
     ]
    }
   ],
   "source": [
    "# token -> words\n",
    "raw_text = raw_text_a + \" \" + raw_text_b\n",
    "text = text_a + \" \" + text_b\n",
    "print(raw_text.split(\" \"))\n",
    "print(tokens)\n",
    "\n",
    "\n",
    "cur_sentence = text.split(\" \")\n",
    "step_output = torch.zeros([len(cur_sentence), 768]).cuda() # +1 to include [SEP]\n",
    "print_tokens = []\n",
    "j = 0\n",
    "\n",
    "for idx, word in enumerate(cur_sentence):\n",
    "#     print(word)\n",
    "    num_tokens = len(tokenizer.tokenize(word))\n",
    "    tmp_tokens = []\n",
    "    for _ in range(num_tokens):\n",
    "        tmp_tokens.append(tokens[j])\n",
    "        step_output[idx, :] += sum_last_four.squeeze()[j]\n",
    "        j += 1\n",
    "    print_tokens.append(tmp_tokens)\n",
    "#     step_output[i,:] /= num_tokens #  you can average the embeddings or not.\n",
    "#     pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([16, 768])\n",
      "[['[CLS]'], ['sea', '##horse', '##s'], ['dreamed'], ['about'], ['word'], ['em', '##bed', '##ding', '##s'], ['and'], ['artificial'], ['intelligence'], ['[SEP]'], ['sea', '##horse'], ['is'], ['a'], ['smart'], ['animal'], ['[SEP]']]\n"
     ]
    }
   ],
   "source": [
    "print(step_output.shape)\n",
    "print(print_tokens)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "salads&pokè\n",
      "['salad', '##s', '&', 'poke']\n"
     ]
    }
   ],
   "source": [
    "print(\"salads&pokè\")\n",
    "print(tokenizer.tokenize(\"salads&pokè\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenizer.convert_tokens_to_ids(tokenizer.tokenize(\"[PAD]\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Confirming contextual feature"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "ename": "IndexError",
     "evalue": "Dimension out of range (expected to be in range of [-1, 0], but got 1)",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mIndexError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-19-7850a251cedc>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m     12\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     13\u001b[0m \u001b[0;32mwith\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mno_grad\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 14\u001b[0;31m     \u001b[0mencoded_layers\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0m_\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmodel\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcontextual_tensor\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;31m# You need the .view(-1, 1)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     15\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     16\u001b[0m \u001b[0mcontextual_encoded_layers\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msum\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstack\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mencoded_layers\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0;36m4\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdim\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/.local/lib/python3.6/site-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, *input, **kwargs)\u001b[0m\n\u001b[1;32m    491\u001b[0m             \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_slow_forward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    492\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 493\u001b[0;31m             \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mforward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    494\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0mhook\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_forward_hooks\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvalues\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    495\u001b[0m             \u001b[0mhook_result\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mhook\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mresult\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/envs/py36/lib/python3.6/site-packages/pytorch_pretrained_bert/modeling.py\u001b[0m in \u001b[0;36mforward\u001b[0;34m(self, input_ids, token_type_ids, attention_mask, output_all_encoded_layers)\u001b[0m\n\u001b[1;32m    728\u001b[0m         \u001b[0mextended_attention_mask\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0;36m1.0\u001b[0m \u001b[0;34m-\u001b[0m \u001b[0mextended_attention_mask\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m*\u001b[0m \u001b[0;34m-\u001b[0m\u001b[0;36m10000.0\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    729\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 730\u001b[0;31m         \u001b[0membedding_output\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0membeddings\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minput_ids\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtoken_type_ids\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    731\u001b[0m         encoded_layers = self.encoder(embedding_output,\n\u001b[1;32m    732\u001b[0m                                       \u001b[0mextended_attention_mask\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/.local/lib/python3.6/site-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, *input, **kwargs)\u001b[0m\n\u001b[1;32m    491\u001b[0m             \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_slow_forward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    492\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 493\u001b[0;31m             \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mforward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    494\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0mhook\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_forward_hooks\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvalues\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    495\u001b[0m             \u001b[0mhook_result\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mhook\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mresult\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/envs/py36/lib/python3.6/site-packages/pytorch_pretrained_bert/modeling.py\u001b[0m in \u001b[0;36mforward\u001b[0;34m(self, input_ids, token_type_ids)\u001b[0m\n\u001b[1;32m    259\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    260\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mforward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minput_ids\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtoken_type_ids\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 261\u001b[0;31m         \u001b[0mseq_length\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0minput_ids\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msize\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    262\u001b[0m         \u001b[0mposition_ids\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0marange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mseq_length\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdtype\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlong\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdevice\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0minput_ids\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdevice\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    263\u001b[0m         \u001b[0mposition_ids\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mposition_ids\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0munsqueeze\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mexpand_as\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minput_ids\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mIndexError\u001b[0m: Dimension out of range (expected to be in range of [-1, 0], but got 1)"
     ]
    }
   ],
   "source": [
    "# BOOM!\n",
    "\n",
    "max_length = 30 # 256/512\n",
    "text = \"After stealing money from the bank vault, the bank robber was seen fishing on the Mississippi river bank.\"\n",
    "contextual_ids = tokenizer.convert_tokens_to_ids(tokenizer.tokenize(\"[CLS] \" + text + \" [SEP]\"))\n",
    "\n",
    "# padding\n",
    "while len(contextual_ids) < max_length:\n",
    "    contextual_ids.append(0)\n",
    "\n",
    "contextual_tensor = torch.LongTensor(contextual_ids).cuda() # dim = [22]\n",
    "\n",
    "with torch.no_grad():\n",
    "    encoded_layers, _ = model(contextual_tensor) # You need the .view(-1, 1)\n",
    "    \n",
    "contextual_encoded_layers = torch.sum(torch.stack(encoded_layers[-4:]), dim=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "max_length = 30 # 256/512\n",
    "text = \"After stealing money from the bank vault, the bank robber was seen fishing on the Mississippi river bank.\"\n",
    "contextual_ids = tokenizer.convert_tokens_to_ids(tokenizer.tokenize(\"[CLS] \" + text + \" [SEP]\"))\n",
    "\n",
    "# padding\n",
    "while len(contextual_ids) < max_length:\n",
    "    contextual_ids.append(0)\n",
    "\n",
    "contextual_tensor = torch.LongTensor(contextual_ids).cuda() # dim = [22]\n",
    "\n",
    "with torch.no_grad():\n",
    "    encoded_layers, _ = model(contextual_tensor.view(1,-1))\n",
    "\n",
    "contextual_encoded_layers = torch.sum(torch.stack(encoded_layers[-4:]), dim=0)    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "30\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "torch.Size([1, 30, 768])"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(len(contextual_ids))\n",
    "contextual_encoded_layers.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 [CLS]\n",
      "1 after\n",
      "2 stealing\n",
      "3 money\n",
      "4 from\n",
      "5 the\n",
      "6 bank\n",
      "7 vault\n",
      "8 ,\n",
      "9 the\n",
      "10 bank\n",
      "11 robber\n",
      "12 was\n",
      "13 seen\n",
      "14 fishing\n",
      "15 on\n",
      "16 the\n",
      "17 mississippi\n",
      "18 river\n",
      "19 bank\n",
      "20 .\n",
      "21 [SEP]\n"
     ]
    }
   ],
   "source": [
    "for i,x in enumerate(tokenizer.tokenize(\"[CLS] \" + text + \" [SEP]\")):\n",
    "    print(i,x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "First fifteen values of 'bank' as in 'bank vault':\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "tensor([ 2.1455, -3.6647, -0.8976, -0.1880,  1.2952,  0.2723, -2.5086,  1.3888,\n",
       "        -0.9232, -2.2783, -0.8348, -1.5500, -0.5243,  1.6353, -3.8010],\n",
       "       device='cuda:0')"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(\"First fifteen values of 'bank' as in 'bank vault':\")\n",
    "contextual_encoded_layers.squeeze()[6][:15]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "First fifteen values of 'bank' as in 'bank robber':\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "tensor([ 1.0651, -2.8878, -0.0540, -0.6728,  0.9880,  1.5694, -2.5486, -0.1006,\n",
       "         0.4420, -1.0535, -0.6334, -0.2961, -0.1209,  2.1677, -3.8774],\n",
       "       device='cuda:0')"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(\"First fifteen values of 'bank' as in 'bank robber':\")\n",
    "contextual_encoded_layers.squeeze()[10][:15]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "First fifteen values of 'bank' as in 'river bank':\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "tensor([ 0.7650, -0.8561, -0.4355, -1.4705,  1.7509, -0.3871,  1.4671,  4.2738,\n",
       "        -0.2546, -1.5692,  2.3346, -0.2058, -0.5221,  1.4032, -3.3482],\n",
       "       device='cuda:0')"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(\"First fifteen values of 'bank' as in 'river bank':\")\n",
    "contextual_encoded_layers.squeeze()[19][:15]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Similarity of 'bank' as in 'bank robber' to 'bank' as in 'bank vault': 0.9276867\n",
      "Similarity of 'bank' as in 'bank robber' to 'bank' as in 'river bank': 0.73005724\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics.pairwise import cosine_similarity\n",
    "# Compare \"bank\" as in \"bank robber\" to \"bank\" as in \"river bank\"\n",
    "different_bank = cosine_similarity(\n",
    "    contextual_encoded_layers.squeeze()[10].view(1,-1).cpu(), \n",
    "    contextual_encoded_layers.squeeze()[19].view(1,-1).cpu())[0][0]\n",
    "\n",
    "# Compare \"bank\" as in \"bank robber\" to \"bank\" as in \"bank vault\" \n",
    "same_bank = cosine_similarity(\n",
    "    contextual_encoded_layers.squeeze()[10].view(1,-1).cpu(),  \n",
    "    contextual_encoded_layers.squeeze()[6].view(1,-1).cpu())[0][0]\n",
    "\n",
    "print(\"Similarity of 'bank' as in 'bank robber' to 'bank' as in 'bank vault':\",  same_bank)\n",
    "print(\"Similarity of 'bank' as in 'bank robber' to 'bank' as in 'river bank':\",  different_bank)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Fine tunning.\n",
    "To make it even better.  \n",
    "https://github.com/huggingface/pytorch-transformers/tree/master/examples/lm_finetuning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### - Input format\n",
    "The scripts in this folder expect a single file as input, consisting of untokenized text, with one sentence per line, and one blank line between documents. The reason for the sentence splitting is that part of BERT's training involves a next sentence objective in which the model must predict whether two sequences of text are contiguous text from the same document or not, and to avoid making the task too easy, the split point between the sequences is always at the end of a sentence. The linebreaks in the file are therefore necessary to mark the points where the text can be split."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "python3 pregenerate_training_data.py \n",
    "--train_corpus yours.txt \n",
    "--bert_model bert-base-multilingual-uncased \n",
    "--do_lower_case \n",
    "--output_dir training/ \n",
    "--epochs_to_generate 3 \n",
    "--max_seq_len 256\n",
    "\n",
    "python3 finetune_on_pregenerated.py \n",
    "--pregenerated_data training/ \n",
    "--bert_model bert-base-multilingual-uncased \n",
    "--do_lower_case \n",
    "--output_dir finetuned_lm  \n",
    "--epochs 3 \n",
    "--reduce_memory \n",
    "--train_batch_size 4\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "stop"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### reference \n",
    "Link to pytorch: https://pytorch.org/  \n",
    "Link to hugging face repo: https://github.com/huggingface/pytorch-transformers  \n",
    "Link to a nicely written tutorial: https://mccormickml.com/2019/05/14/BERT-word-embeddings-tutorial/  \n",
    "Link to code for further classification models: https://colab.research.google.com/drive/1ywsvwO6thOVOrfagjjfuxEf6xVRxbUNO#scrollTo=DEfSbAA4QHas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
